Queues and Messages

Foopipes is built around message queues and topic based messaging. It has a built in memory based message queue, or can be configured to use an external durable message queue such as RabbitMQ.

pipelines: 
  -
    when: 
      - {service: webhook }
    then:
      - {task: publish, service: queue, topic: build }
  -
    when: 
      - {service: queue, topic: build }
    do:
      - {task: command, command: "bash", args: "build.sh" }
    then: 
      - {task: publish, service: queue, topic: push }

Built in Queue service

Foopipes has a built in memory based message queue service with name queue. You don't need to configure this service, but it can be replaced with a different queue of choice.

plugins:
  - MyQueuePlugin
services:
  queue:
    type: myqueueplugin
    url: ampq://myhost/

System messages topics

• started an event message fired when a Foopipe instance has been initialized and started.

pipelines:
  - 
    when: 
      - { service: queue, topic: started } 
    from: 
      - { task: httpget, url: "https://registry.npmjs.org/foopipes"} 
    do: 
      - { task: select, path: "$.versions.*" } 
    to: 
      - { task: file, filename: "foopipe_#{version}.json" } 

Publish/await task results

One tricky feature of Foopipes is that a pipeline can forked into a different pipeline right after a do task. It can also put the pipeline on hold until the fork returns.

Under the hood this is accomplished using messages and replies.

pipelines:
  - 
    when: 
      - { service: queue, topic: started } 
    from: 
      - { task: httpget, url: "https://registry.npmjs.org/foopipes"} 
    do: 
      - { task: select, path: "$.versions.maintainers[*]", publish: email_maintainer, await: false, keepResult: false } 
      - { task: select, path: "$.versions.*" } 
    to: 
      - { task: file, filename: "foopipe_#{version}.json" } 
  - 
    when: 
      - { service: queue, topic: email_maintainer } 
    do: 
      - { task: module, module: emailer, function: sendMail, name: "#{name}", email: "#{email}" }